Preview Deployments (01b5219)

Deployment status

CI has deployed the worker + dashboard to dev (PR-28 stage):

• Worker: cloudflare-metrics-api-dev-pr-28 with */5 * * * * cron trigger
• Grafana dashboard: Cloudflare Account Overview uploaded to the cloudflare-metrics (pr-28) folder in dev Grafana

The worker is currently no-op on every cron tick because the analytics API token secret isn't wired up yet — TF_VAR_cloudflare_analytics_api_token is commented out in deployment/.env (see commit 6154677) and the worker defaults to an empty string, logging a cron_error{reason="missing_config"} self-metric.

Follow-up to start data collection

To make the collector start emitting data to VictoriaMetrics/Grafana:

1. Create a new Cloudflare API token in the dashboard with only the Account Analytics Read permission group, scoped to the target account.
2. Store it in 1Password as CLOUDFLARE_METRICS_ANALYTICS_TOKEN in both the tf_dev and tf_prod vaults.
3. Uncomment the line in deployment/.env:

   export TF_VAR_cloudflare_analytics_api_token="op://tf_$ENVIRONMENT/CLOUDFLARE_METRICS_ANALYTICS_TOKEN/password"
   

4. Re-run CI (push any commit). The next cron tick within 5 minutes will start emitting cf_* metrics.

Why the token isn't generated via Terraform

The cleanest approach would be resource "cloudflare_api_token" { ... } with a hardcoded permission group UUID. I tried that (commit 65e0cf2) but the Terraform service account token doesn't have the User → API Tokens → Write permission required to call POST /user/tokens — returned 403. Both the data-source lookup and the resource creation hit the same endpoint. Getting this fully automated would need either (a) granting the Terraform service account the user-token-write permission, or (b) a dedicated bootstrap token stored out-of-band.

Integration test coverage

pnpm run test:integration is gated on CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID and walks every dataset against the live API. Local run against the prod account: 20/20 datasets succeeded, 11,330 metric points emitted across a 1h window. Once the dev token is in place the same tests can run in CI.

✅ Pipeline is now live end-to-end

Terraform now owns the API token

Following the devtools api-keys pattern, the analytics token is provisioned via cloudflare_api_token using a second aliased provider (cloudflare.bootstrap) that authenticates with the user-level var.cloudflare_api_token instead of the account-scoped token. No manual 1Password step required.

One wrinkle: Cloudflare provider v5 has a bug (#5045) where cloudflare_api_token.value is only populated immediately after creation and gets wiped on refresh. To work around it, there's a terraform_data.analytics_token_generation trigger — bumping its input forces the token to be destroyed and recreated on the next apply, which re-populates .value in the same plan/apply cycle, and the worker binding picks up the fresh value before the worker_version is committed.

The real bug

What took most of the debugging: my first version of the GraphQL client had private readonly fetchImpl: typeof fetch = fetch as a constructor default. In the Workers scheduled runtime that default captured undefined, so every fetchDataset call blew up with a synchronous TypeError before ever reaching the network. The worker was running through all 20 datasets in ~290ms, pushing 20 error metrics + summary to VictoriaMetrics, and that was it — no GraphQL calls at all.

Fix: look up globalThis.fetch lazily at the call site (9303520).

Verified working

Latest cron tick at 16:55:52 UTC:

• subrequests: 22 (20 to api.cloudflare.com + 1 metric flush + 1 leftover diagnostic beacon, now removed)
• cpuTimeUs: 98053, wallTime: 13.3s
• api.cloudflare.com: 20 requests, all 200 OK, 617 KB of response body
• VictoriaMetrics flush: 609 KB request body, 204 OK

That's the full 20-dataset collection making it end-to-end. The dev Grafana Cloudflare Account Overview dashboard in the cloudflare-metrics (pr-28) folder should start showing data within the next couple of cron cycles.

✅ Resource name enrichment (D1, queues, zones)

4822b0c + 9d8d8e8 deployed. At the start of each cron tick, the collector now fetches:

• GET /accounts/{id}/d1/database → database_name tag on cf_d1_* metrics
• GET /accounts/{id}/queues → queue_name tag on cf_queue_* metrics
• GET /zones?account.id={id} → zone_name tag on cf_http_requests_overview metrics

Both the *_id / *_tag and the new *_name tags are emitted, so existing queries keep working and the human-readable name is a strict addition. The Grafana dashboard now groups by name with id as a secondary label.

If any of the three lookups fails it's reported via a cloudflare_metrics_resource_lookup{resource,status,error} self-metric; collection still proceeds with whatever caches are populated.

Token permissions

The cloudflare_api_token now includes D1 Read and Queues Read alongside Account Analytics Read. Permission group UUIDs are looked up dynamically via data.cloudflare_api_token_permission_groups_list on the cloudflare.bootstrap provider — that token already has user-level permissions so the lookup works. Zones list doesn't need a dedicated permission.

Verified on 17:30:52 cron tick

• subrequests: 24 → 20 GraphQL + 3 REST (D1/queues/zones) + 1 VictoriaMetrics flush
• All 23 api.cloudflare.com calls returned 200
• CPU 100ms, wall 15.1s

Dashboard legends should now show real names in dev Grafana within the next cycle.

Zones: per-tag lookup for Pages projects

Root cause: GET /zones?account.id=X only returns the 6 registered zones. The 17 distinct zoneTags in httpRequestsOverviewAdaptiveGroups include ~15 Cloudflare Pages project zones (e.g. immich-app-archive.pages.dev) which aren't in the bulk list but ARE resolvable via GET /zones/{id}.

Fix

• CloudflareRestClient.getZone(zoneId) added to fetch a single zone by id (returns null on 404 so we can tolerate deleted zones).
• CloudflareMetricsCollector.resolveMissingZones(rows) runs after fetching the HTTP overview data: it extracts any zoneTag not already in the cache, fetches each in parallel with Promise.allSettled, and emits a cloudflare_metrics_resource_lookup{resource="zones_individual",status,requested,resolved,failed} self-metric.
• applyResourceTags now falls back to zone_name = zoneTag when the lookup failed entirely, so dashboard legends are never empty.
• Token gained Zone Read in a second policy block scoped to com.cloudflare.api.account.zone.*. (First attempt nested the resource under the account scope and Cloudflare returned "Partial wildcard scope can only be followed by a match-all object expression" — fixed by using "*" directly.)

Verified on the 17:55 tick

• 41 calls to api.cloudflare.com — all 200 OK (was 23 × 200 + 20 × 403 before)
• 20 GraphQL + 3 bulk REST + 17 individual zone lookups + 1 metric flush = 41
• All cf_http_requests_overview rows now carry zone_name — either the real name for Pages zones or the bulk-listed account zones, or the zoneTag as a last resort

Dashboard legends should now show real hostnames like immich-app-archive.pages.dev within the next cron cycle.

✅ 1-minute granularity live, batched, and working

Final results on the 20:25:52 cron tick:

New datasets shipped

• cf_d1_queries_detail — per-query counts/rows/duration (p50/p95/p99) from d1QueriesAdaptiveGroups, grouped by database_id/database_role/error
• cf_queue_consumer — consumer concurrency_avg per queue from queueConsumerMetricsAdaptiveGroups
• cf_http_requests_detail — per-zone detailed HTTP breakdown (count, bytes, visits, timing) from httpRequestsAdaptiveGroups grouped by country/method/status/cache/protocol (zone-scoped)
• cf_workers_scheduled — cron-specific invocation stats from workersInvocationsScheduled, aggregated client-side into (script, cron, status, minute) buckets with invocations/cpu_time sum/avg/max

firewallEventsAdaptiveGroups is excluded — gated on Business/Enterprise plans and returns authz/"does not have access to the path" at both account and zone scopes on our plan. A comment in datasets.ts documents how to add it if we upgrade.

Granularity

Dropped from datetimeFiveMinutes to datetimeMinute across all 24 datasets. That's the finest grouping the Cloudflare Analytics API exposes — going below 1-minute would require the raw datetime sample stream, which is not aggregated and would explode cardinality. The collector window widened to 12 minutes (still overlapping the previous run by 7 minutes so one missed cron doesn't lose data).

Batching architecture

Three design changes in graphql-client.ts:

1. fetchAccountBatch — all account-scope datasets that share a filter granularity are emitted as aliased fields under one viewer.accounts query. workersInvocationsScheduled rides along as a workers_scheduled alias in the datetime batch. Partial responses (one field erroring while others succeed) are preserved via executeAllowPartial + groupErrorsByAlias.

2. fetchZoneBatch — all bulk-listed zones for a zone-scoped dataset get aliased zones(filter: {zoneTag: "..."}) blocks in a single query. Zone tags are validated against /[a-zA-Z0-9_-]+/ before being inlined to block injection.

3. globalZoneNameCache module-level map persists Pages zone name lookups across Worker isolate invocations. Cold-start costs ~20 /zones/{id} lookups (throttled); warm isolates skip them entirely.